TCPIP Network Stack Performance in Linux Kernel 2.4 and 2.5

We discuss our findings on how well the Linux 2.4 and 2.5 TCPIP stack scales with multiple network interfaces and with the SMP network workloads on 100/1000 Mb Ethernet networks. We identify three hotspots in the Linux TCPIP stack: 1) inter-processor cache disruption on SMP environments, 2) inefficient copy routines, and 3) poor TCPIP stack scaling as network bandwidth increases. Our analysis shows that the L2 cache_lines_out rate (thereby memory cycles per instructionmCPI) is high in the TCPIP code path leading to poor SMP Network Scalability. We examine a solution that enhances data cache effectiveness and therefore improves the SMP scalability. Next the paper concentrates on is improving the “Copy_To_User” and “Copy_From_User” routines used by the TCPIP stack. We propose using the “hand unrolled loop” instead of the “movsd” instruction on the IA32 architecture and also discuss the effects of aligning the data buffers. The gigabit network interface scalability workload clearly shows that the Linux TCPIP stack is not efficient in handling high bandwidth network traffic. The Linux TCPIP stack needs to mimic the “Interrupt Mitigation” that network interfaces adopt. We explore the techniques that would accomplish this effect in the TCPIP stack. This paper also touches on the system hardware limitation that affects the gigabit NIC’s scalability. We show that three or more gigabit NICs do not scale in the hardware environment used for the workloads.